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United  States  General  Accounting  Office 
Washington,  DC  20548 


September  5,  2001 

The  Honorable  Dana  Rohrabacher 
Chairman,  Subcommittee  on  Space 
and  Aeronautics 
Committee  on  Science 
House  of  Representatives 


The  Honorable  Bart  Gordon 
Ranking  Minority  Member,  Subcommittee 
on  Space  and  Aeronautics 
Committee  on  Science 
House  of  Representatives 

Subject:  Survey  of  NASA’s  Lessons  Learned  Process 


You  asked  us  to  assess  whether  the  National  Aeronautics  and  Space  Administration 
(NASA)  has  adequate  mechanisms  in  place  to  ensure  that  past  lessons  learned  from 
mission  failures  are  being  applied.  As  part  of  your  request,  you  asked  that  we  provide 
the  results  of  our  survey  regarding  the  collection  and  sharing  of  lessons  learned  by 
NASA  program  and  project  managers.  This  letter  provides  the  results.  We  plan  to 
issue  a  more  detailed  report  on  the  effectiveness  of  NASA’s  lessons  learned  process 
early  next  calendar  year. 

As  part  of  our  assessment  of  NASA’s  effectiveness  in  applying  lessons  learned,  we 
surveyed  all  of  NASA’s  program  and  project  managers  to  obtain  their  perspectives  on 
(1)  the  collection,  access,  and  use  of  lessons  learned,  (2)  the  strengths  and  limitations 
of  current  NASA’s  lessons  learned  processes,  procedures,  and  systems,  including  the 
Lessons  Learned  Information  System  (LLIS),  (3)  the  challenges  or  barriers  to  the 
sharing  of  lessons  learned,  and  (4)  suggested  areas  of  improvement.  This 
represented  192  managers  overseeing  about  240  programs  and  projects.  On  August  6 
and  8,  2001,  we  briefed  your  staff  on  the  results  of  our  survey  and  this  report 
summarizes  those  results.  The  briefing  slides  are  included  as  enclosure  I. 

We  conducted  our  work  from  September  2000  through  July  2001  in  accordance  with 
generally  accepted  government  auditing  standards.  Enclosure  II  describes  our 
objectives,  scope  and  methodology. 
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Abstract 

You  asked  us  to  assess  whether  the  National  Aeronautics  and  Space  Administration  (NASA)  has  adequate 
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your  request,  you  asked  that  we  provide  the  results  of  our  survey  regarding  the  collection  and  sharing  of 
lessons  learned  by  NASA  program  and  project  managers.  This  letter  provides  the  results.  We  plan  to  issue 
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BACKGROUND 


NASA’s  procedures  and  guidelines  require  that  program  and  project  managers  review 
and  apply  lessons  learned  from  the  past  throughout  a  program’s  or  project’s  life  cycle 
and  to  document  and  submit  any  significant  lessons  learned  in  a  timely  manner. 

NASA  defines  a  lesson  learned  as  “knowledge  or  understanding  gained  by  experience. 
The  experience  may  be  positive,  as  in  a  successful  test  or  mission,  or  negative,  as  in  a 
mishap  or  failure.  A  lesson  must  be  significant  in  that  it  has  a  real  or  assumed  impact 
on  operations;  valid  in  that  it  is  factually  correct;  and  applicable  in  that  it  identifies  a 
specific  design,  process,  or  decision  that  reduces  or  limits  the  potential  for  failures 
and  mishaps,  or  reinforces  a  positive  result.”  The  failure  of  the  Mars  Polar  Lander 
and  Climate  Orbiter  spacecraft  in  1999  raised  concern  that  lessons  learned  from  past 
mishaps  and  programs  were  not  being  effectively  applied  towards  future  mission 
success. 

Lessons  learned  systems  are  used  by  many  military,  commercial  and  government 
organizations  to  capture,  store,  disseminate,  and  share  knowledge  gained  from  past 
experiences.  NASA’s  principal  mechanism  for  collecting  and  sharing  lessons  learned 
from  programs,  projects,  and  missions  agency  wide  is  the  LLIS.  The  goal  of  LLIS  is  to 
ensure  that  NASA  does  not  have  to  keep  “relearning”  the  lessons  of  the  past.  NASA 
also  shares  lessons  learned  through  revisions  to  its  policies  and  guidance.  Further, 
lessons  learned  from  a  mishap  or  operational  event  are  captured  in  procedure  and 
process  documents. 

SURVEY  RESULTS 


Our  survey  highlighted  fundamental  weaknesses  in  the  collection  and  sharing  of 
lessons  learned  in  NASA  by  program  and  project  managers  as  well  as  in  the  agency’s 
LLIS.  While  some  lessons  learning  does  take  place,  our  survey  identified  that  lessons 
are  not  routinely  identified,  collected,  or  shared  by  program  and  project  managers.  In 
addition,  many  respondents  indicated  that  they  are  dissatisfied  with  NASA’s  lessons 
learned  processes  and  systems.  Respondents  also  identified  challenges  or  barriers  to 
the  sharing  of  lessons  learned  as  well  as  areas  of  improvement.  Specifically, 
respondents  indicated  the  following. 

•  Limited  Sharing  of  Lessons  Learned  Agency-wide 

Lessons  learned  are  of  little  value  unless  they  are  collected  and  shared  with  others. 

To  determine  the  extent  that  lessons  are  being  collected  and  shared  within  NASA,  we 
asked  program  and  project  managers  how  knowledgeable  they  were  of  lessons 
generated  by  their  own  programs  and  centers  as  well  as  by  other  centers.  Program 
and  project  managers  responded  that  they  are  very  or  generally  knowledgeable  of 
lessons  generated  by  their  own  programs  and  centers  but  are  not  very  knowledgeable 
of  lessons  generated  by  programs  at  other  centers.  This  fact  is  supported  by  survey 
results  that  show  managers  primarily  identify  lessons  through  program  or  center- 
based  activities  such  as  project  reviews  or  informal  discussions  with  colleagues. 

LLIS  was  not  identified  as  a  primary  source  for  lessons  learning. 
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•  LLIS  Is  Not  a  Primary  Mechanism  for  Collecting  and  Sharing  Lessons 
Learned 


Our  survey  identified  several  reasons  why  LLIS  is  not  the  primary  mechanism  used  by 
program  and  project  managers  for  collecting  and  sharing  of  lessons.  One  reason  is  a 
lack  of  awareness  about  the  system.  About  one-third  of  program  and  project 
managers  responded  that  they  first  learned  of  LLIS  from  our  questionnaire  or  only 
became  aware  of  the  system  within  the  last  6  months.  This  level  of  awareness  also 
corresponds  with  survey  results  that  show  limited  use  of  LLIS  by  program  and  project 
managers  within  the  last  2  years. 

•  Dissatisfaction  with  Lessons  Learned  Processes  and  Systems 

Our  survey  also  determined  that  more  managers  are  very  or  generally  dissatisfied 
with  NASA’s  lessons  learned  processes  and  systems,  including  LLIS,  than  are 
satisfied.  This  level  of  dissatisfaction  may  stem  from  the  fact  that  58  percent  of 
managers  stated  that  current  processes  and  systems  do  not  allow  them  to  retrieve  the 
right  lessons  at  the  right  time.  Another  contributing  factor  to  this  level  of 
dissatisfaction  could  be  because  program  and  project  managers  do  not  find  existing 
lessons  helpful:  53  percent  of  managers  said  they  found  lessons  learned  useful  less 
than  25  percent  of  the  time. 

•  Barriers  Exist  to  Lessons  Learning 

If  lessons  learning  is  to  succeed  it  is  important  that  an  organizational  culture  exist 
that  fosters  the  value  of  sharing  knowledge  based  on  others’  experiences.  Our  survey 
determined  that  many  managers  do  not  believe  that  implementing  lessons  learned 
significantly  improves  mission  effectiveness.  One  manager  stated  that  “Until  we  can 
adopt  a  culture  that  admits  frankly  to  what  really  worked  and  didn’t  work,  I  find 
many  of  these  tools  to  be  suspect.” 

Another  critical  barrier  highlighted  by  the  survey  is  the  lack  of  time  available  for 
lessons  learned.  One  manager  noted  that  “  It  is  time  consuming  to  agree  on  correct 
lessons  learned  for  a  project,  accurately  describe  the  issue. .  .in  a  way  that  is  helpful 
to  a  project.”  Finally,  many  managers  noted  in  their  narrative  responses  that  there  is 
reluctance  to  share  negative  lessons  for  fear  that  they  might  not  be  deemed  to  be 
good  project  managers.  According  to  some  respondents  this  appears  to  stem  from  a 
culture  that  sees  lessons  learned  as  negative,  i.e.,  an  admission  of  failure. 

•  Suggested  Improvements  to  Lessons  Learning 

While  program  and  project  managers  identified  a  number  of  ways  for  improving 
lessons  learning  within  NASA,  mentoring  was  their  first  choice.  One  manager  noted 
that  the  most  effective  lessons  learned  process  results  from  the  passing  down  of 
lessons  from  experienced  people  to  those  less  experienced.  Other  suggestions  for 
improvement  included  “storytelling”  or  “post  mortems.”  A  manager  stated  that  story 
telling  would  be  a  quick  way  to  get  the  message  distributed  and  allow  persons  to  ask 
questions.  Another  notable  suggestion  was  that  senior  management  be  accountable 
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for  the  infusion  or  engineering  of  lessons  into  a  project  with  some  kind  of  a 
matrix/metric  to  validate  that  the  lessons  was  actually  incorporated. 

Managers  also  suggested  a  number  of  improvements  regarding  LLIS.  Suggested 
improvements  included  (1)  inclusion  of  more  positive  lessons  learned,  (2)  an 
improved  search  capability,  (3)  addition  of  a  frequently  asked  question  section,  and 
(4)  addition  of  information  to  allow  one  to  determine  whether  a  lesson  is  applicable 
to  a  specific  project.  Another  suggestion  was  that  lessons  be  distributed  once  a 
month  and  an  alert  used  to  notify  managers  about  important  lessons. 

AGENCY  COMMENTS 


NASA  generally  agreed  with  our  draft  report.  Our  efforts  to  assess  the  effectiveness 
of  the  NASA’s  lessons  learning  processes  and  procedures  is  continuing  and  we  plan 
to  issue  a  final  report  on  the  results  of  our  work  early  in  the  next  calendar  year. 
NASA’s  comments  in  their  entirety  appear  in  enclosure  III. 

We  are  sending  copies  of  this  letter  to  the  NASA  Administrator  and  interested 
congressional  committees.  The  letter  is  also  available  on  GAO's  home  page  at 
http://www.gao.gov.  If  you  or  your  staff  have  any  questions  regarding  the  briefing  or 
this  letter,  please  contact  me  at  (202)  512-4841,  or  John  Oppenheim,  Assistant 
Director  at  (202)  512-3111.  Key  contributors  to  this  assignment  were  James  A.  Elgas, 
Carl  M.  Ramirez,  Jose  A.  Ramos,  Lorene  S.  Same,  and  Diana  Dinkelacker. 

Sincerely  yours, 


Allen  Li 
Director 

Acquisition  and  Sourcing  Management 
Enclosures  -  3 


Page  4 


GAO-01-1015R  Survey  of  NASA’s  Lessons  Learned  Process 


ENCLOSURE  I 
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Slides  from  the  Briefing  on  the  Results 
of  NASA’s  Lessons  Learned  Survey 
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NASA  Lessons  Learned 
Survey  Results 


Briefing  to  the 
House  Science 
Committee,  Subcommittee 
on  Space  and  Aeronautics 


August  6,  2001 
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Purpose  of  Survey 


•  Obtain  views  of  National  Aeronautics  and  Space 
Administration  (NASA)  program  and  project  managers 
regarding: 

•  Collection,  sharing,  and  application  of  lessons  learned 

•  Strengths  and  limitations  of  NASA's  lessons  learned 
processes,  procedures,  and  systems,  including  NASA's 
Lessons  Learned  Information  System  (LLIS) 

•  Challenges  or  barriers  to  the  sharing  of  lessons  learned 

•  Suggested  areas  of  improvement 
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Survey  Scope 


•  Conducted  an  online  survey  of  1 92  program/project  managers 
in  June/July  2001 

•  Covered  all  Enterprise  areas: 

•  Earth  Science 

•  Human  Exploration  and  Development  of  Space 

•  Aerospace  Technology 

•  Space  Science 

•  Biological  &  Physical  Research 

•  Identified  244  programs  and/or  projects 
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Survey  Instrument  Development 


Obtained  relevant  information  from  interviews  with  agency  officials 
involved  in  program  and  project  management 


Reviewed  lessons  learned  related  documents  and  guidance 


•  Analyzed  literature  on  lessons  learned  and  knowledge 
management 
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ENCLOSURE  I 


•  NASA  Headquarters  and  others  reviewed  and  commented  on  the 
proposed  contents  of  the  questions 

•  Conducted  a  pretest  of  the  survey  questions  with  program  and 
project  managers  at  Goddard  Space  Flight  Center  to  gauge  clarity, 
and  length  of  time  of  survey  administration 

•  Names/addresses  of  NASA  program  and  project  managers  were 
provided  by  NASA  at  our  request 

•  We  believe  that  our  survey  results  are  generalizable  to  the 
population  of  NASA  program  and  project  managers 

•  Practical  difficulties  of  conducting  any  survey,  however,  may 
introduce  errors  into  estimates  made  from  surveys 
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NASA’s  Definition  of  Lessons  Learned 


“Knowledge  or  understanding  gained  by  experience.  The 
experience  may  be  positive,  as  in  a  successful  test  or  mission,  or 
negative,  as  in  a  mishap  or  failure.  A  lesson  must  be  significant  in 
that  it  has  real  or  assumed  impact  on  operations;  valid  in  that  it  is 
factually  and  technically  correct;  and  applicable  in  that  it  identifies  a 
specific  design,  process,  or  decision  that  reduces  or  limits  the 
potential  for  failures  and  mishaps,  or  reinforces  a  positive  result.” 
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NASA  Processes,  Procedures,  and  Systems  for 
Lessons  Learning 


NASA’s  Program  and  Project  Management  Processes 
and  Requirements  (NPG  7120.5A)  states  that  program 
and  project  managers: 


•  should  review  and  apply  lessons  learned  from  the 
past  throughout  the  program/project  life  cycle,  where 
appropriate 


•  shall  document  and  submit  any  significant  lessons 
learned  in  a  timely  manner 


•  NASA’s  Procedures  and  Guidelines  for  mishap 
investigating  (NPG  8621.1)  requires  the  identification 
and  submission  of  lessons  learned 
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NASA  Has  Established  Processes,  Procedures, 
and  Systems  for  Lessons  Learning 


•  NASA’s  Lessons  Learned  Information  System  (LLIS)  is 
an  on-line,  automated  data  base  system,  was 
established  to  facilitate  the  capture  and  sharing  of 
lessons  learned  agency-wide 


•  NASA  disseminates  lessons  through  periodic  updates 
to  policies  and  guidance 

•  Lessons  learned  through  mishaps  or  other  operational 
events  are  incorporated  into  procedure  and  process 
documents 
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Characteristics  of  Survey  Respondents 


•  Sixty  (60)  percent  response  rate 

Respondent  Demographics  Respondents  by  Enterprise  Area 
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Summary  of  Survey  Results 

I 


The  survey  identified  fundamental  weaknesses  and 
improvements  in  the  collection  and  sharing  of  lessons 
learned: 

•  Limited  sharing  of  lessons  learned  agency-wide 

•  LLIS  is  not  the  primary  mechanism  for  collecting  and 
sharing  lessons  learned 

•  There  are  barriers  to  lessons  learning 

•  Dissatisfaction  with  lessons  learned  processes  and 
procedures 

•  Suggested  improvements  to  strengthen  lessons  learned 
processes  and  procedures 
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Limited  Sharing  of  Lessons  Learned 
Agency-wide 

•  Program/project  managers  are  knowledgeable  of  lessons 
learned  in  their  own  programs/centers,  but  are  less 
knowledgeable  of  lessons  at  other  centers 


EH 

No  Answer 

□ 

Not  Ve  ry 

K  now  ledge  a  ble 

Somewhat 

K  now  ledge  a  ble 

□ 

Very  Knowledgable 
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Limited  Sharing  of  Lessons  Learned 
Agency-wide 

•  No  systematic  method  for  alerting  program  and  project 
managers  of  important  lessons 
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•  Sharing  of  lessons  mostly  occurs  during  project  review  and 
on  an  informal  basis 
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LLIS  is  Not  the  Primary  Mechanism  for 
Collecting  and  Sharing  Lessons  Learned 


Principal  sources  for  identifying  lessons  learned: 

•  1st  Source:  Systems  and  Engineering  Reviews 

•  2nd  Source:  Program/Project  Briefings 

•  3rd  Source:  Informal  Discussions  with  Colleagues 

Twenty-seven  (27)  percent  of  program  and  project  managers 
were  not  aware  of  LLIS  before  the  survey 

Only  17.5  percent  of  the  respondents  have  applied  a  lessons 
learned  identified  through  LLIS 

In  the  past  2  years  only  23  percent  of  program  and  project 
managers  have  submitted  a  lessons  learned  to  LLIS 
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LLIS  is  Not  the  Primary  Mechanism  for 
Collecting  and  Sharing  Lessons  Learned 

•  In  the  last  two  years  43  percent  of  program  and  project 
managers  have  not  submitted  a  lesson  to  LLIS 
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LLIS  is  Not  the  Primary  Mechanism  for 
Collecting  and  Sharing  Lessons  Learned 

•  LLIS  is  seldom  used  by  programs  and  projects  to  look  for 
lessons  over  the  past  two  years 


No  answer 
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Barriers  Exist  to  Lessons  Learning 


Thirty-nine  (39)  percent  of  program/project  managers 
believe  that  implementing  lessons  learned  contributes  only 
some  or  little  to  improving  mission  effectiveness 


Managers  believe  there  is  too  little  time  for  individuals  to 
share  lessons  learned: 


“It  is  time  consuming  to  agree  on  correct  lessons  learned  for  a 
project,  accurately  describe  the  issue  and  the  lesson  in  a  way 
that  is  helpful  to  a  project” 


•  Managers  are  reluctant  to  share  failures  with  others: 


[There  is  an]  unwillingness... to  share  information  or  air  dirty 
laundry.  If  you  made  a  mistake  you  might  not  be  deemed  to 
be  a  good  project  manager.” 
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Dissatisfaction  with  Lessons  Learned 
Processes  and  Systems 

•  More  respondents  are  very  or  generally  dissatisfied  with 
NASA’s  processes  and  systems  than  are  satisfied 

•  Fifty-eight  (58)  percent  stated  that  processes  and  systems 
infrequently  allows  them  to  retrieve  the  right  lessons  at  the 
right  time 


•  Managers  indicate  that  not  enough  emphasis  is  being  placed 
on  reporting  positive  or  successful  experiences 
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Suggestions  Provided  by  Managers  to 
Improve  Lessons  Learning  at  NASA 


Mentoring: 


“I  believe  ‘lessons  learned’  [equals]  experience. 
Experience  is  gained  on  the  job,  and  much  of  it  is 
passed  along  informally  through  mentoring.  This 
means  there  must  be  a  base  of  experienced  people 
who  must  be  allowed  to  mentor  the  newer 
employees!” 


•  Expanded  practice  of  “storytelling”  or  “post  modems” 
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Suggestions  Provided  by  Managers  to 
Improve  Lessons  Learning  at  NASA 


Develop  a  capability  to  disseminate  or  notify  users  about 
important  lessons  learned: 


“Centers  should  transmit  public  (Center-wide)  emails 
containing  important  or  prominent  lessons  learned  on 
a  regular  basis” 


_ jtGAO 

Suggestions  Provided  by  Managers  to 
Improve  Lessons  Learning  at  NASA 

•  Better  integration  of  lessons  learned  into  program/project 
planning  and  implementation  procedures 


•  Allow  more  time  for  the  lessons  learned  process: 


“If  it  [lessons  learned]  is  important  then  make  it 
important.  Give  it  the  processes,  procedures, 
resources,  and  particularly  [the]  time,  to  make  it  a 
positive  experience.  However,  merely  making  it 
another  requirement  to  an  already  overwhelming  work 
schedule  will  doom  it  to  the  failure  it  presently  may  be 
experiencing” 
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is 


Recognition  of  the  Importance  of 
Lessons  Learned 


GAO 


“I  am  encouraged  by  the  focus  and  attention  the  lessons  learned 
process  is  getting.  I  really  hope  that  the  follow-through  is 
sustained. ..rhetoric  is  cheap  and  gets  the  ‘motherhood  and  apple 
pie’  salute.  The  real  action  of  process  development  and 
improvement  [is  done]  by  actually  engineering  the  lessons  into  the 
processes. ..[but  this]  is  costly  and  hard.  A  lot  of  the  work  is  about 
culture  change  to  accept  a  new  model  that  contrasts  to  ‘the  way 
we've  always  done  it.’" 


Project  Manager 
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Survey  Objectives.  Scope  and  Methodology 


To  obtain  the  views  of  NASA  program  and  project  managers  on  lessons  learned 
processes  and  systems  at  NASA  we  conducted  a  survey  in  June  and  July  2001, 
using  a  self-administered  electronic  questionnaire  posted  on  the  World  Wide  Web. 
The  survey  contained  four  groups  of  questions  on  (1)  how  NASA  personnel 
collect,  access  and  use  lessons  learned;  (2)  the  strengths  and  weaknesses  of 
current  NASA  lessons  learned  processes,  procedures  and  systems,  including  the 
Lessons  Learned  Information  System;  (3)  potential  challenges  or  barriers  to 
sharing  Lessons  Learned  within  NASA;  and  (4)  ways  to  improve  the  lessons 
learned  process  or  system  at  NASA. 

In  designing  the  questionnaire,  we  interviewed  NASA  officials,  program  and 
project  managers,  as  well  as  other  government  and  industry  officials  who  had 
insight  into  lessons  learned  processes  and  systems.  We  also  obtained  and 
reviewed  NASA  documents  and  guidance  pertinent  to  lessons  learned.  To  further 
guide  the  development  of  appropriate  questions,  we  reviewed  current  literature 
on  lessons  learned  and  knowledge  management.  To  validate  the  content  and 
structure  of  the  questionnaire,  we  submitted  it  to  officials  at  NASA  Headquarters 
and  others  for  review  and  incorporated  their  comments  where  appropriate.  To 
verify  the  clarity,  length  of  time  of  administration,  and  suitability  of  the  questions, 
we  also  pretested  the  questionnaire  with  selected  program  and  project  managers 
at  Goddard  Space  Flight  Center. 

NASA  officials  provided  us  with  a  list  of  the  e-mail  addresses  of  199  personnel 
that  they  indicated  were  representative  of  all  NASA  program  and  project 
managers  as  of  April  2001.  On  June  5,  2001,  we  sent  e-mail  messages  to  each 
person  in  this  survey  population  notifying  them  of  the  survey  and  asking  them  to 
complete  the  questionnaire.  Subsequently,  we  discovered  one  additional  NASA 
manager  who  was  eligible  for  our  survey,  and  removed  7  managers  who  were 
ineligible  for  the  survey  because  they  were  not  project  or  program  managers  at 
the  time  of  the  survey.  We  also  eliminated  one  duplicate  listing,  resulting  in  a  final 
survey  population  of  192. 

Over  the  following  several  weeks  until  closing  the  survey  on  July  13,  2001,  we 
received  a  total  of  115  useable  responses,  for  an  overall  response  rate  of  60 
percent.  All  of  the  responses  had  been  submitted  using  the  web  questionnaire, 
although  one  participant  had  asked  to  submit  a  paper  version  of  the  survey,  due  to 
problems  in  accessing  the  web  survey.  Of  the  77  non-respondents,  9  provided 
partial  questionnaire  responses  but  had  not  indicated  that  they  were  finished  with 
the  questionnaire. 

While  we  believe  that  our  survey  results  are  generalizable  to  the  population  of 
NASA  program  and  project  managers  as  described  above,  the  practical  difficulties 
of  conducting  any  survey  may  introduce  errors  into  estimates  made  from  surveys. 


Page  16 


GAO-01-1015R  Survey  of  NASA’s  Lessons  Learned  Process 


ENCLOSURE  II 


ENCLOSURE  II 


Although  we  administered  questionnaires  to  all  known  members  of  the 
population,  and  thus  our  results  are  not  subject  to  sampling  error,  nonresponse  to 
the  entire  survey  or  individual  questions  can  introduce  a  similar  type  of  variability 
or  bias  into  our  results  to  the  extent  that  those  not  responding  differ  from  those 
who  do  respond  in  how  they  would  have  answered  our  survey  questions.  In 
addition,  population  coverage  errors  can  occur  if  some  members  of  the 
population  are  excluded  from  the  survey.  Measurement  errors  can  arise  from 
how  questions  are  interpreted  by  respondents  and  mistakes  made  by  respondents. 
Data  processing  errors  can  arise  during  the  handling  or  analysis  of  responses.  We 
took  steps  in  the  design,  data  collection  and  analysis  phases  of  our  survey  to 
minimize  such  errors,  such  as  pretesting  questionnaires  before  the  survey, 
following  up  with  those  not  reachable  at  original  e-mail  addresses  or  otherwise 
not  immediately  responding,  and  checking  for  errors  in  computer  programming 
used  to  analyze  survey  results.  In  addition,  the  distribution  of  respondents  across 
NASA  Enterprise  areas  generally  reflected  the  actual  distribution  of  the  entire 
population,  which  was  consistent  with  our  belief  that  nonresponse  error  was  not 
significant. 
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Comments  From  the  National  Aeronautics 


National  Aeronautics  and 
Space  Administration 

Office  of  the  Administrator 
Washington,  DC  20546-0001 


Mr,  Allen  Li  AUS  2  9  2001 

Director 

Acquisition  and  Sourcing  Management 
U.  S.  General  Accounting  Office 
Washington,  DC  20548 


Dear  Mr.  Li: 

We  appreciate  the  opportunity  to  review  and  provide  comments  to  the  draft  report, 
entitled,  Survey  of  NASA’s  Lessons  1  earned  Process.  We  have  reviewed  the  report  in 
detail  and  arc  essentially  in  agreement  with  the  survey  results  and  suggested  improvements. 

We  continue  to  evaluate  NASA  processes  and  will  do  the  same  for  lessons  learned. 
Major  revisions  to  NASA  Procedures  and  Guidelines  (NPG)  7120.5,  NASA  Program  and 
Project  Management  Processes  and  Requirements,  are  in  the  final  stages  and  include  more 
clarity  in  the  use  and  scope  of  the  Lessons  Learned  Information  System  (LLIS).  Since 
NASA  uses  numerous  sources  for  lessons  learned,  LLIS  will  no  longer  be  referred  to  as  the 
primary  source,  but  one  of  many. 

Programs  and  projects  benefit  from  lessons  learned  day-to-day  in  many  different 
forms.  Some  of  these  include  status  reporting,  programmatic  and  technical  reviews,  and 
functional  staff  meetings.  The  more  formal  and  widespread  forms  include  technical 
standards  and  updating  NPG’s  and  other  documentation.  The  various  NASA  training 
programs  also  incorporate  lessons  learned  into  their  curricula,  including  storytelling  by 
current  and  retired  NASA  program  and  project  managers.  One  forum  is  designed 
specifically  for  sharing  lessons  learned  and  is  titled  Project  Management  Shared  Experiences 
Program.  The  overall  objective  of  this  program  is  to  provide  a  forum  to  understand  key 
initiatives  influencing  NASA  project  management  and  for  project  people  to  share 
knowledge,  experiences,  and  creative  approaches  to  project  management. 

NASA  must  do  a  better  job  of  communicating  the  various  lessons  learned  sources, 
improving  mechanisms  by  which  these  various  sources  are  linked,  and  encouraging  the 
proper  training  for  employees  in  order  to  maximize  the  ability  of  current  activities  to  draw 
upon  the  lessons  of  others.  We  plan  continued  improvement  in  these  areas.  Should  you 
have  any  questions,  please  call  Keith  Hudkins,  Deputy  Chief  Engineer,  at  202-358-1823. 


Sincerely, 


Daniel  R.  Mulvillc 
Associate  Deputy  Administrator 


(707565) 
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